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Abstract. Effects of synaptic noise on the retrieval process of associative memory 
neural networks are studied from the viewpoint of ncurobiological and biophysical 
understanding of information processing in the brain. We investigate the statistical 
mechanical properties of stochastic analog neural networks with temporally fluctuating 
synaptic noise, which is assumed to be white noise. Such networks, in general, defy the 
use of the replica method, since they have no energy concept. The self-consistent signal- 
to-noise analysis (SCSNA), which is an alternative to the replica method for deriving 
a set of order parameter equations, requires no energy concept and thus becomes 
available in studying networks without energy functions. Applying the SCSNA to 
stochastic network requires the knowledge of the Thouless- Anderson-Palmer (TAP) 
equation which defines the deterministic networks equivalent to the original stochastic 
ones. The study of the TAP equation which is of particular interest for the case 
without energy concept is very few, while it is closely related to the SCSNA in the case 
with energy concept. This paper aims to derive the TAP equation for networks with 
synaptic noise together with a set of order parameter equations by a hybrid use of the 
cavity method and the SCSNA. 
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1. Introduction 

The replica method p] for random spin systems has been successfully employed in neural 
network models of associative memory [21 E]. However the replica method requires the 
concept of free energy. On the other hand, various types of neural network models 
which have no energy concept, such as a network with asymmetric synaptic coupling 
or temporally fluctuating synaptic coupling, may be in existence. The self-consistent 
signal-to-noise analysis (SCSNA) jUGSE], which is an alternative approach to the replica 
method for deriving a set of order parameter equations, requires no energy concept. 
Thus it can be applicable to study the statistical properties of wider class of networks 
including networks without energy concept. 

The SCSNA, which was originally proposed for deriving a set of order parameter 
equations for a deterministic analog neural network with or without energy functions, 
becomes applicable to a stochastic network by noting that the TAP equation defines 
the deterministic one equivalent to the original stochastic one. The SCSNA is closely 
related to the Thouless- Anderson- Palmer (TAP) equation [TJ [8] via the concept of cavity 
method in the case where a network has energy concept [6] and the relationship between 
the two was studied in detail in the networks with two-body and multi-body interactions 
[SI E] . The coefficient of the Onsager reaction term characteristic to the TAP equation 
which determines the form of the transfer function is self-consistently obtained through 
the concept shared by the cavity method and the SCSNA. 

The TAP equation is of our interest for studying the statistical properties of a 
network without energy concept. However the TAP equations for a network with 
synaptic noise are not found in literatures. The main target of this paper is to derive the 
TAP equation for a network with temporally fluctuating synaptic noise as multiplicative 
noise. 

The effects of such synaptic noise on the retrieval properties of networks have been 
studied in some recent works [101 [UJ [T21 H3J. According to the stochastic resonance 
theory [HI [15l US] , the temporally fluctuating synaptic noise may possibly be expected 
to reduce the interference of the uncondensed patterns on the retrieval property of the 
network (noise in terms of stochastic resonance) and, as a result, enhance the retrieval 
property or the storage capacity (signal) of the network. However such an argument is 
not found in literatures. To discuss the effect of temporally fluctuating synaptic noise 
on the retrieval property, it is important to construct a tractable model to study the 
role played by such synaptic noise. 

In literatures, the term "synaptic noise" is used in three meanings: (i) quenched 
disorder in synaptic couplings [32], (ii) randomness related to the dilution of synaptic 
couplings [18] , and (iii) temporal fluctuation of synaptic couplings [TUl EH 121 13] . We 
will use the term "synaptic noise" in the third meaning in the present paper. 

Cortes et al [22] (see also [13]) investigated the case where neurons obey a master 
equation with continuous time and the synaptic couplings obey (i) slow dynamics 
PH12Q], (ii) fast dynamics pH H31 [2H 122] (iii) middle speed dynamics [23 [21] compared 
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to the dynamics of neurons. In the first case, since the synaptic couplings obey slow 
dynamics, the adiabatic approximation for the synaptic couplings becomes exact in the 
limit where the time scale of synapse dynamics r — > oo and quenched random noise in 
couplings again arises. Thus the synaptic noise is regarded as the well-known quenched 
random variable and this type of synaptic noise has been studied as "synaptic noise" in 
many literatures [25] . 

In the second case, since the dynamics of the synaptic noise is sufficiently fast 
compared to the dynamics of neurons, one can define the effective strength of synaptic 
coupling by averaging the temporally fluctuating synaptic coupling [101 [TTJ [121 131 126] . 
In this case, one can find the effective Hamiltonian of the network and use well-known 
replica method |lj to obtain the order parameters analytically [21 [3]. Other example 
of the fast synaptic dynamics can be found in [27], which studies the properties of the 
equilibrium state of the system with stochastically evolving couplings. 

On the other hand, the third case is difficult to deal with analytically especially 
in the case where the number of memory patterns is proportional to the total number 
of neurons and only numerical results based on computer simulations exist [23, 24J. 
In spite of these recent efforts to elucidate the effects of synaptic noise on the 
retrieval properties of neural networks, such preceding studies have been based on the 
macroscopic viewpoint, where the order parameters solely have been investigated, and 
the TAP equations for such cases have not been reported. 

The purpose of this paper is two-fold: (i) we will derive the TAP equation for 
a stochastic analog network with temporally fluctuating multiplicative synaptic noise 
which is not found in literatures, (ii) We will study the SCSNA and the TAP equation 
for such a network to elucidate the effects of the multiplicative synaptic noise on the 
retrieval property from both microscopic and macroscopic viewpoint. Part of this work 
is reported elsewhere [28] . 

This paper is organized as follows: in the next section, we will describe an analog 
neural network model with temporally fluctuating synaptic noise as multiplicative noise 
which is assumed to be white noise to write down a set of Langevin equations, and derive 
the corresponding Fokker-Planck equation. We will see that the equilibrium solution 
of the Fokker-Planck equation is given as a Gibbs probability density with the effective 
temperature, which should be determined self-consistently in the thermodynamic limit. 
In section 3, we will apply the cavity method to derive the formal expression of the TAP 
equation (pre- TAP equation). Then using the SCSNA, we will self-consistently obtain 
the concrete form of the transfer function which yields the complete form of the TAP 
equation as well as a set of order parameter equations. In section 4, the phase diagram 
for our model will be shown. In the last section, we will conclude this paper. 
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2. Model and Fokker-Planck equation formalism 

Let us deal with the following stochastic analog neural network of iV neurons with 
temporally fluctuating synaptic noise: 

±i = -<j>'{Xi) + ^ J ij$) x 3 + > ( la ) 

(Vi(t)vj(t')) =2D8 ij 5(t-t r ), (lb) 

where Xi (i = 1, ■ • ■ , N) represents a state of the neuron at site % taking a continuous 
value, 6( potential of an arbitrary form which determines the probability 

distribution of in the case without the input X^-(^i) Jij x j> Vi the Langevin white noise 
with its noise intensity 2D and Jij(t) the synaptic coupling. We note here that, in the 
case of associative memory neural network, the synaptic coupling is usually defined 
by the well-known Hebb learning rule. However some experimental results show that 
the synaptic couplings have temporal fluctuations which originate from the dynamics of 
neurotransmitters or kinetics of ion channels independent of that of neurons [29], and 
hence the effects of such synaptic noise may be relevant to the retrieval properties in 
realistic networks. To investigate such effects of synaptic noise, we assume the synaptic 
coupling taking the form: 

Jij(t) = Jij + eijit) , (2a) 
2D 

(e l] (t)e kl (t')) = —SikSjtfit - t') , (26) 

where is defined by the usual Hebb learning rule = ^^ =1 £f £^ with p = aN the 
number of patterns embedded in the network, £f = ±1 is the /z th embedded pattern at 
neuron i, and €ij(t) denotes the synaptic noise independent of rji(t), which we assume in 
our model as white noise with its intensity 2D/N for simplicity. Notice that, in equation 
(JTaj) . the synaptic noise behaves as multiplicative noise and the synaptic coupling Jij(t) 
is asymmetric. 
Noting 

I rt+At rt+At I \ 2£) 

by means of Ito integral, we obtain the Fokker-Planck equation corresponding to the 
Langevin equation ffTaj) as 



&P(t,x) A d I JJI . ^ - /_ ~\ d 



dt 




+ E ■'■r'-, ~ (D + Dq) — \ P(t, x) , (3) 

where q = ^ Ylij(^i) x2 j- Since the self-averaging property holds in the thermodynamic 
limit iV — > oo, one can identify q as 
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where (•) represents the thermal average with respect to P(t,x). Thus equation ([3]) 
is found to be a nonlinear Fokker-Planck equation whose diffusion coefficient D + Dq 
depends on the probability density P(t, x) [30j US]- In this paper we are concerned with 
deriving the TAP equation and order parameter equations for the equilibrium state self- 
consistently. Furthermore the order parameter q is also obtained self-consistently in our 
framework as seen below. Supposing q is given, the Fokker-Planck equation ([3]) turns 
to be a linear equation and one can easily find the equilibrium probability density for 
the linear Fokker-Planck equation (J3j) as 

Pat(x) = Z' 1 exp j-/3 cff ^2 4>( x i) ~ Z~2 Jii x i x ^j J > ( 5 ) 

where Z denotes the normalization constant and 

(3 J = D + Dq (6) 

plays the role of the effective temperature of the network. Notice that the temperature of 
the system is modified to (3~q as a consequence of the multiplicative noise and it depends 
on the order parameter q. Here it is easily checked that the equilibrium distribution of 
the system becomes Gibbs distribution in the thermodynamic limit iV — > oo. 

Since we have explicitly written down the equilibrium probability distribution 
density (JSJ) as a form of Gibbs distribution, one can define the (effective) Hamiltonian 
of iV-body system as 

N 

H N = 4>(Xi) - 22 Jij X i X 3 ■ ( 7 ) 

i=l i<j 

Then regarding the original network with multiplicative noise ([Taj) as an analog version 
of the standard Hopfield model whose Hamiltonian is given by equation (CO) with the 
effective temperature one can apply the usual cavity method [2] to this system and 
derive the (pre-) TAP equation. 



3. Cavity method and self-consistent signal-to-noise analysis 

We have obtained the equilibrium probability density as a form of Gibbs distribution 
(JSJ) and the effective Hamiltonian ((Tj) in the previous section. Thus the cavity method 
[2], which is usually applied to the network models for deriving the TAP equation, is 
applicable for our model. According to the cavity method, we divide the Hamiltonian 
of iV-body system (0) into that of (N — l)-body system and the part involving in the 
state of 2 th neuron as 

H N = <p(xi) - hiXi + H N _ X , 

where hi = Jij x j * s the local field at site i and the Hamiltonian of (N — l)-body 

system if^-i is given as i/^-i = 4>{ x j) ~ J2j<k(^i) Jjk x j x k- Then the marginal 
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probability density distribution of %i and the local field hi is given as 



P N (x i} hi) 



Y[ dxj 5 | hj - JijXj Pat(x) 

= Z~ l exp { -peg [<f>{xi) - hiXi}} P N _i{hi) , 

where Z is the normalization constant and Pjv-i(^i) denotes the probability density of 
the local field hi in the (N — l)-body system defined as 



PN-i{hi 



7-1 

J N-1 



n d% i 









hi 



l3 Xj 



exp [-fiesH 



N-H 7 



where Z^-i denotes the normalization constant. Since the local field is given as the 
summation of a sufficiently large number of random variables and their cross-correlations 
are expected to be 0(1/ \/N), one can expect that P^-i(^i) turns out to be a Gaussian 
density in the thermodynamic limit N — > oo according to the central limit theorem: 



P N -i(h) 



V2 



exp 



(hj - (hj) N -if 
2a 2 



where (-)jv-i represents the thermal average with respect to the (A^— l)-body probability 
density P/v_i(x) and a 2 is the variance of Pjv-i(^i)' which is evaluated later self- 
consistently in the framework of the SCSNA. Then taking the average of Xj with respect 
to the marginal probability P^(xi, hi) straightforwardly yields 

(x i )=F((h i ) N - 1 ), (8) 

where F is a transfer function defined as 



J dxxexp < —(3 e s 4>(x) 



yx 

/•(//) 

J dx exp < — (3 eS cj)(x) — yx — 
Similarly (/ij)jv-i is obtained as 

(hi) N -i = (hi) - p efi a 2 (xi) . 
Thus we have the pre- TAP equation 



geffO _ x 2 



(9) 



{X; 



i) = F\ ^Jij&jj-ToM , (10) 

\3&i) ) 

where r 0ns = PefiO~ 2 . Since the concrete form of the transfer function F depends on the 
effective temperature /3 e g and the variance of the local field a 2 , it is necessary to obtain 
P e s and a 2 to have the TAP equation [5, 6j. 

Equation ffTOj) is regarded as defining a deterministic analog network corresponding 
to the original stochastic one (JTaj) . and hence we can apply the SCSNA to equation ffTOl) 
to determine P e g and <r 2 self-consistently as was studied for the case without synaptic 
noise [5j [6]. For simplicity, we here assume that the only one condensed pattern j^ 1 } 
is retrieved. The extension to the case of an arbitrary finite number of condensed 



TAP equation for analog neural network with temporally fluctuating synaptic noise 7 

patterns is straightforward. Using the overlap order parameter m M = £f (%i), 

the equilibrium average of the local field is rewritten as 

(h i )=dm 1 + Y l ^-a(x i ). (11) 

Using the SCSNA, the above local field can be rewritten as [5, 6J 

{hi) = + $W + z ifl + r SCS NA(x i ) , (12) 

where J2v>2 £i mU = £f TO/ * + z %y. + l{ x i)i Tscsna = 7 ~ ot. and z ifM is a Gaussian random 
variable with zero mean. As seen below, we will evaluate the overlap m M self-consistently, 
and then obtain z^ and 7 through the equivalence between the expression of the local 
field (JTT1) and ( 1T21) . Substituting equation (Tl2l) into the pre- TAP equation fTTOT) reads 

( Xi ) = F (tfm 1 + + + (T SCSNA - T ns)(x l )) 

and comparing this equation with equation (JSj) yields [6] 

TsCSNA = Tons , 

since (hi) n-i is considered to be a Gaussian random variable which should not contain 
the Onsager reaction term. Noting that m M = 0(l/ViV) for fi > 2, one can obtain the 
overlap for uncondensed patterns as 

N 



N(l-U) ._ s 



^H^+Zj,), (13a) 



1 - 

U =_5^F'(gm 1 + % M ), (136) 
i=i 

where F' denotes the derivative of the transfer function F and the order expansion of F 
with respect to 1/VN has been applied to (xi) = F(Qm} + z iM + ^fm M ). Using equation 
(I13aj) and the definitions of and 7, one finds 

a 

7 = 



1 3(&) 



Thus the variance of z iM is evaluated as 



2 



1 -C/p< F2 ^ ml+ ^)^' ( 14a ) 

where (•)$,*: represents the average over a random variables £ = ±1 and the Gaussian 
variable 2, and the self-averaging property has been used. Similarly one obtains the set 
of order parameter equations as 

m l = {iF(im l + z)) iz , (146) 
U=(F'^m 1 + z)) £z , (14c) 



Ons — J- SCSNA 



u 



(Ud) 
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In the case where the multiplicative synaptic noise does not exist or the intensity of the 
synaptic noise is zero, i.e., (3 e ^ — (3 = 1/D, the set of order parameter equations (I14aj) . 
( 1146j) . (I14cp . ( 114rfl) takes a closed form and determines the form of the transfer function F 
as well as the order parameters self-consistently. For the case with multiplicative noise, 
however, it does not suffice to determine the form of the transfer function. We need the 
order parameter q, which determines /? e g, as well as m 1 , U, a 2 , r 0ns to determine the 
concrete form of F. Since q is related to the macroscopic susceptibility of the system 
and, by definition of F (02), the order parameter U corresponds with the susceptibility 
as U = Pes((x 2 ) — (x) 2 ), one finds 

. U (1 -U) 2 2 , 1A , 

q = - r + - L cr 2 z . 14e 



The set of equations (jSJ), (114 aft . (114&j) . f|14c|) . (114dj) . (114 ep takes a closed form and thus 



one can determine the form of F self-consistently as well as the set of order parameters. 
Therefore substituting into the pre- TAP equation (fit)]) the solutions /3 e g and Tons that 
are self-consistently obtained within this framework yields the TAP equation. 

4. Phase diagram and numerical results 

We have derived the TAP equation as well as the set of order parameter equations in 
the previous section. In this section we show the phase diagram by solving the set of 
order parameter equations (114 aft . ( 11461) . ( 114cl) . (114rfl) . (114el) numerically and investigate 
the effect of the multiplicative synaptic noise. 

For the well-known transfer function of the Ising neurons F(x) = tanh(/5x), it is 
easy to understand the effects of the interference of the synaptic noise. This choice of 
the transfer function is equivalent to taking the potential as 

exp[-/W(:r)] _l s{x _ 1)+ l s{x+1)m 



J exp [— ^ e g<p(x)] dx 2 2 

In the Ising neuron model, since q is simply given as q = 1 and (3~^ = D+D, the retrieval 
state vanishes for D > 1 according to the results of Amit-Geutfreund-Sompolinsky 
(AGS) [3J. 

In this section, for simplicity, we consider the double-well potential whose minima 
are located at x = ±1: 

4>{x) = jx* - ^x 2 , (15) 

where A determines the depth of the wells of the potential. This potential yields a 
continuous distribution of neuron states and thus defines an analog network model in 
which q is non-trivial. We investigate a phase diagram for the analog network model 
and elucidate the effects of the multiplicative noise on the retrieval properties. 

Figure [1] illustrates the storage capacity a as a function of the intensity of the 
external noise D. The solid line is for the absence of the synaptic noise, i.e., D — 0. 
The line in figure [1] denotes the numerical solution of the order parameter equations for 
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Figure 1. Storage capacity a as a function of the intensity of the external noise 
D for various values of D. The solid curve denotes the storage capacity for D = 0. 

The broken ( ), dashed ( ) and dotted ( ) curves represent the storage 

capacity for D = 0.2, 0.4, 0.6, respectively. The retrieval state locates below the curve 
and vanishes at D ~ 1.05. We set A = 20. 

each intensity of the synaptic noise D = 0, 0.2, 0.4, 0.6 and the retrieval state vanishes at 
D ~ 1.05 for A = 20. The a-D line for the analog network is deformed compared to the 
Ising networks. This is the effect of potential properties and the effective temperature, 
or the non-trivial order parameter q, while q = 1 for Ising neurons. 

We can see that the storage capacity is incrementally decreased as the intensity of 
the synaptic noise increases. This result is reasonable since the memories are encoded 
in the synaptic coupling as local minima of the effective free energy corresponding to 
equation (j3j) and the synaptic noise is expected to disturb the fine structure of the energy 
landscape. 

Figure [2] displays the a-dependence of the overlap m 1 obtained from the SCSNA 
together with that from numerical simulations with iV = 3000. We can see that the 
overlap m 1 decreases as the number of embedded patterns a increases and retrieval state 
vanishes (m 1 = 0) at a c ~ 0.049. 

Figure [3] illustrates the distribution of the thermal average of the state of neurons, 
or the local magnetization (xA at D — 0, D — 0.5, a = 0.1, A = 20 obtained from 
numerical simulations with N = 3000. We can see from figure [2] that the overlap 
m 1 = in this regime. However the local magnetizations (xj)'s are seen to distribute 
around (xA = ±1. This means that a spin glass phase arises in this regime. We can 
also show the existence of the non-retrieval spin glass phase analytically by solving the 
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Figure 2. a-dependence of the overlap m 1 obtained from the SCSNA (solid curve) 
together with that from numerical simulations with N = 3000 (dots). The potential 
is given by equation (fT5|) with A = 20. The intensities of additive and multiplicative 
synaptic noise are D = and D — 0.5 respectively. 




-1.5 -1 -0.5 0.5 1 1.5 

local magnetization 



Figure 3. Distribution of the thermal average of the state of neurons, or the local 
magnetization with parameters D — 0, D — 0.5, a = 0.1, A = 20. From figure [2] 
this parameter set locates in the non-retrieval phase. This distribution shows that the 
non-retrieval spin glass phase arises in this regime. 
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set of order parameter equations (jBD, f)14ap . (1146j) . (114 cj) . (114dj) and (114 ep . Since m 1 = 
in the non-retrieval phase, the order parameter equations f 1 1 4 g[) and (j 14 eft become 

U=(F'(z)) z , (166) 

where (-) 2 denotes the average with respect to the Gaussian random variable z. For 
the non- retrieval phase m 1 = 0, it is trivial that the order parameter equation 
m 1 = = (C,F(z))^ z holds. By definition of the transfer function F, the order parameter 
U is rewritten as 

U = Mq-q), (16c) 

where q = ^Yli( x i) 2 * s the Edward- Anderson order parameter. The set of order 
parameter equations flSj), fll4dj) - (114 ep . (I16al) . (|166p . (1 16 el) takes a closed form. Thus we 



can find the non-retrieval spin glass phase by solving these equations to obtain q ^ 0. 
Since the Edward-Anderson order parameter q is expected to be small in the regime 
close to the paramagnetic phase, the Taylor expansion with respect to q is applicable 
for these order parameter equations to illustrate the paramagnetic (q = 0)-spin glass 
(q 0) phase boundary. This boundary is expected to correspond to the de Almeida- 
Thouless (AT) line. The study on the relationship between the SCSNA and the replica 
symmetry breaking is underway. 



5. Concluding remarks 

We have derived the TAP equation for a stochastic analog neural network with 
temporally fluctuating multiplicative synaptic noise, which is not found in literatures. 
More specifically, we have derived the TAP equation together with the set of order 
parameter equations by using the SCSNA and the cavity method. Our original model 
does not have the concept of free energy. Since the self-averaging property holds in 
the thermodynamic limit iV — > oo, we have found that the nonlinear Fokker-Planck 
equation (jHJ) becomes quasi-linear to allow one to obtain equilibrium probability density 
obeying the Gibbs one with effective temperature and hence that the network with 
white synaptic noise has the effective Hamiltonian in the large N limit. Thus the cavity 
method, which is applicable to the model with energy concept, becomes available to 
obtain the (pre-)TAP equation. Unlike the case without synaptic noise, the concrete 
form of the transfer function F of our model has been found to depend not only on the 
coefficient of the Onsager reaction term but on the order parameter q. q as well as the 
coefficient of the Onsager reaction term have been obtained self-consistently within the 
framework of the SCSNA. The full TAP equation straightforwardly follows from the 
pre- TAP equation by substituting the solutions of the order parameter equations into 
the pre- TAP equation (TTU1) . 

Furthermore, we have found that the storage capacity of the network gradually 
decreases as the intensity of the synaptic noise increases, since the fine structure of the 
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energy landscape tends to disappear by the interference of the synaptic noise. This 
effect of the interference of the synaptic noise on the behavior of the retrieval property 
has been shown to appear via the effective temperature (3~^ > D. 

All the results presented in this paper are obtained via the cavity method and 
the SCSNA. On the other hand, the order parameter equations (114 aft . ( 11461) . (1 1 4 cl) . 
fl!4dj) . (114 ej) can be reproduced as replica symmetric case by the replica method, since 
the system has the effective Hamiltonian ([7]). Thus our results are expected to be 
exact within the replica symmetric approximation. However, the development of the 
analysis in the framework of the SCSNA for replica symmetry breaking solutions is now 
underway. 

In other works dealing with the temporal fluctuation in synaptic couplings [TTl [12] . 
the authors study the case of the fast synapse dynamics. Then the synaptic coupling is 
modified to take the form of "effective synaptic coupling" and the system becomes 
to have an effective Hamiltonian. In this case the "effective synaptic coupling" is 
straightforwardly determinedby both the number of embedded patterns and the intensity 
of Langevin noise associated with neuron dynamics. On the other hand, the time 
scale of fluctuation of synaptic coupling in our model is comparable to that of the 
neuron dynamics. Our model results in having the "effective temperature" and hence 
the effective Hamiltonian in the thermodynamic limit. However, in our model, the 
"effective temperature" is determined only self- consistently together with the other order 
parameters. 

In this paper we have dealt with a network subjected to asymmetric multiplicative 
synaptic noise given as white noise involving both pre- and post-neuron and the noise 
has no correlation with the synaptic coupling given by the Hebb learning rule. However 
some other versions of synaptic noise may be considered: (i) synaptic noise depending 
only on pre- or post-neuron, (ii) synaptic noise correlated with the Hebb learning rule, 
(iii) colored synaptic noise. For some of these cases, one can rigorously derive the TAP 
equation and the set of order parameter equations similarly to the case we have seen in 
this paper. The analysis for such cases will be reported elsewhere. 
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